Notes on Single-Pass Online Learning Algorithms

Online learning methods are typically faster and have a much smaller memory footprint than batch learning methods. However, in practice online learners frequently require multiple passes over the same training data in order to achieve accuracy comparable to batch learners. We investigate the problem of single-pass online learning, i.e., training only on a single pass over the data. We compare the performance of single-pass online learners to traditional batch learning, and we propose a new modification of the Margin Balanced Winnow algorithm that can reach results comparable to linear SVM for several NLP tasks. We also explore the effect of averaging, a.k.a. voting, on online classifiers. We provide experimental evidence that voting can be successfully used to boost the performance of several single-pass online learning algorithms. Finally, we describe how the Modified Margin Balanced Winnow algorithm proposed can be naturally adapted to perform online feature selection. This scheme performs comparably to information gain or chi-square, with the advantage of being able to select features on-the-fly.